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HOMOGRAPHY TRANSFER FROM POINT MATCHES 



TECHNICAL FIELD OF THE INVENTION 



The present invention is directed, in general, to view 
morphing for three dimensional video image generation and, 
more specifically, to use of homography transfer and 
epipolar geometry for computing homographies in extended 
image sequences . 



A homography is a linear projective transformation 
relating two images for either of two cases: (1) a planar 
world object projected onto image planes for the two 
images, where the image planes relate to arbitrary camera 
motion; or (2) the projection of arbitrary world images 
onto image planes for pure camera rotation only. A pinhole 
camera model illustrated in FIGURE 4A is employed for the 
first view to define the location within the camera 
reference frame (x , y , and z, where z is the optical axis) 
of an image point m representing the intersection of a line 
containing a three dimensional world point W and the 
camera's optical center C with an image plane R . The 



BACKGROUND OF THE INVENTION 




DOCKET NO. 70^V6 (PHIL06-02596) PATENT 



projection of the optical center C for a similar pinhole 
fcamera representing the second view onto the image plane R 
of the first camera or view, illustrated in FIGURE 4B, is 
epipole e, and epipole e' is similarly the projection of the 
5 optical center C onto the image plane R' of the second 

camera or view. 

The image points m (defined by coordinates x,y ) and m r 
(defined by coordinates x\y r ), which represent the 
projection of the image pixel W onto the image planes R 

a 

10fi and R f , respectively, constitute a conjugate pair, and a 

yQ 

fundamental matrix F relating such conjugate pairs may be 
computed (see, e.g., u Epipolar Geometry and the Fundamental 
Matrix" in R. Hartley & A. Zisserman, Multiple View 
Geometry in Computer Vision, pp. 219-242 (Cambridge 



nf 



1SJ1 University Press 2000) , incorporated herein by reference) 

^ and employed to develop a 3X3 homography matrix H {2 for 

projective mapping of image points from the first camera or 
view reference frame to the second according to the linear 
proj ective equation : 
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which may be equivalently written as 
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h n x + h n y + h x 
h 3l x + h 32 y + h 3 
h 2l x + h 22 y + h, 
h 3l x + h n y + k 
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Homography describes a one-to-one point mapping between the 
two images for either pure rotation by the camera or for 
world image points within a plane. The homography matrix 
for the plane at infinity H Uqo , which maps vanishing points 
to vanishing points, is an important case and depends only 
on the rotational component of the rigid displacement. 

Computation of homography is a very important and 
difficult task in, for example, three dimensional (3D) 
video generation and rendering. This is especially true 
for computation of the homography matrix for the plane at 
infinity, which requires knowledge of either (1) three sets 
or parallel lines, (2) point matches between the views 
corresponding to points very far away from the camera, or 
(3) complete camera calibration parameters. None of these 
is easy to compute for the general case. 

There is, therefore, a need in the art for a 
simplified method of computing the homography matrix for 
the plane at infinity, particularly in extended image 
sequences . 
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SUMMARY OF THE INVENTION 



To address the above-discussed deficiencies of the 
prior art, it is a primary object of the present invention 
to provide, for use in video system, transfer of an 
infinity homography for an image pair within an image 
sequence to other image pairs within the image sequence 
utilizing point matches for the subject image pairs. An 
image set including the image pair for which the infinity 
homography is known and a third image are selected. Then 
intermediate parameters for infinity homography transfer 
for one image pair overlapping the known infinity 
homography image pair are computed from the known infinity 
homography and epipoles and fundamental matrices of the 
overlapping image pairs, to derive the infinity homography 
for the selected overlapping image pair. The process may 
then be repeated until infinity homographies for all image 
pairs of interest within the image sequence have been 
derived. 

The foregoing has outlined rather broadly the features 
and technical advantages of the present invention so that 
those skilled in the art may better understand the detailed 
description of the invention that follows. Additional 
features and advantages of the invention will be described 
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hereinafter that form the subject of the claims of the 
invention. Those skilled in the art will appreciate that 
they may readily use the conception and the specific 
embodiment disclosed as a basis for modifying or designing 
other structures for carrying out the same purposes of the 
present invention. Those skilled in the art will also 
realize that such equivalent constructions do not depart 
from the spirit and scope of the invention in its broadest 
form. 

Before undertaking the DETAILED DESCRIPTION OF THE 
INVENTION below, it may be advantageous to set forth 
definitions of certain words or phrases used throughout 
this patent document: the terms "include" and "comprise," 
as well as derivatives thereof, mean inclusion without 
limitation; the term "or" is inclusive, meaning and/or; the 
phrases "associated with" and "associated therewith," as 
well as derivatives thereof, may mean to include, be 
included within, interconnect with, contain, be contained 
within, connect to or with, couple to or with, be 
communicable with, cooperate with, interleave, juxtapose, 
be proximate to, be bound to or with, have, have a property 
of, or the like; and the term "controller" means any 
device, system or part thereof that controls at least one 
operation, whether such a device is implemented in 
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hardware, firmware, software or some combination of at 



functionality associated with any particular controller may 
be centralized or distributed, whether locally or remotely. 
Definitions for certain words and phrases are provided 
throughout this patent document, and those of ordinary 
skill in the art will understand that such definitions 
apply in many, if not most, instances to prior as well as 
future uses of such defined words and phrases. 



least two of the same. 



It should be noted that the 
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BRIEF DESCRIPTION OF THE DRAWINGS 



For a more complete understanding of the present 
invention, and the advantages thereof, reference is now 
made to the following descriptions taken in conjunction 
with the accompanying drawings, wherein like numbers 
designate like objects, and in which: 

FIGURE 1 depicts a block diagram of a video system 
employing a method of homography transfer utilizing point 
matches according to one embodiment of the present 
invention; 

FIGURE 2 depicts image data employed in computing 
homography transfer utilizing point matches according to 
one embodiment of the present invention; 

FIGURE 3 is a high level flowchart for a process of 
computing homography transfer utilizing point matches 
according to one embodiment of the present invention; and 

FIGURES 4A and 4B depict models for computing epipolar 
geometry and homography. 
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DETAILED DESCRIPTION OF THE INVENTION 



FIGURES 1 through 3, discussed below, and the various 
embodiments used to describe the principles of the present 
invention in this patent document are by way of 
illustration only and should not be construed in any way to 
limit the scope of the invention. Those skilled in the art 
will understand that the principles of the present 
invention may be implemented in any suitably arranged 
device . 

FIGURE 1 depicts a block diagram of a video system 
employing a method of homography transfer utilizing point 
matches according to one embodiment of the present 
invention. Video system 100 includes a video receiver, 
which may be a computer or other data processing system; a 
satellite, terrestrial or cable broadcast television 
receiver; a set top box; or a video game unit. Video 
receiver 101 has an input 103 for receiving a sequence of 
images and an optional output 104 for transmitting 
processed image data to, for example, a display or 
recording device . 

Video receiver 101 in the exemplary embodiment 
includes an image processor 104 computing homography 
transfer using point matches for the received sequence of 
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images as described in further detail below. 



Video 



deceiver 101 may also optionally include a memory 105 for 



storing the received sequence of images, computed 
homography information, or other data, and a display 106 
for displaying images or image sequences modified utilizing 
the computed homography information. For example, image 
processor may be employed in depth reconstruction, body 
modeling, or image morphing for three dimensional (3D) 
television, and may generate altered image sequences based 
on received image sequences to display a scene from a 
different perspective. 

Those skilled in the art will recognize that the 
complete structure and operation of a video receiver is not 
depicted or described herein. Instead, only so much of the 
structure and operation of a video receiver as is either 
unique to the present invention or necessary for an 
understanding of the present invention is depicted in the 
figures and described. The remainder of the structure and 
operation of a suitable video receiver may be conventional, 
although in the present invention the video receiver 
computes homography information for the received image 
sequence as described in further detail below. 

FIGURE 2 depicts image data employed in computing 
homography transfer utilizing point matches according to 
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one embodiment of the present invention. A sequence 2 00 of 



integer) is received by image processor 104. Also received 
by image processor 104, or alternatively, computed by image 
processor 104, is a tabulation 201 of point matches between 
any given pair of images within the image sequence 2 00 and 
a homography 202 at the plane of infinity (or "infinity 
homography") for two consecutive images within the image 
sequence 2 00 . 

Without loss of generality, the infinity homography 
202 is assumed to be for the first two consecutive images Ii 
and I 2 within image sequence 2 00, and homography transfer 
according to the present invention will be described for 
the first three images: I 1# I 2 and I 3 . Since point matches 
between all three images are known, computation of epipoles 
e t j and fundamental matrices F~ between any image pair within 

the set of images I 1# I 2 and I 3 is straightforward, and may 
be performed using any of the applicable methods known in 
the art . . 

As known in the art, arbitrary homographies (for an 
arbitrary plane z) between two images i,j are related to 

epipoles e g and fundamental matrix for those images by 

the expression: 



images l lf I 2/ I3, 



I n (where n is any positive nonzero 



Hi 
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F y =[e^H y (1) 
where H & is a rank 2 homography and [^] x is derived from 
the epipolar vector e u by: 

0 -e 3 e 2 
e 3 0 -e { 
-e 2 e x 0 

Homography H jt is similarly related to epipoles e }i and 
fundamental matrix F }i . Since epipoles e (j and e y . , funda- 
mental matrices F & and F jt , and homographies # (> . and H jt are 
defined only up to a scale, an additional constraint that 



hi- 



gp each of these vectors/matrices have a norm equal to 1 is 

KJlj introduced. 
M 3 It is also known in the art (although not very widely 

employed) that an arbitrary homography H tj between two 
images may be written as: 



H.=c'.H-. +e-.z' 1 (2) 
y y y y v ; 

15 where c\j is an unknown scalar, H ija0 is the infinity homo- 

graphy mapping image points from image i to corresponding 
image points within image j , and z[J is the transpose of an 
unknown vector. Equation (2) may be equivalently written: 

H^=c 9 H ¥ +e g zl (3) 
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where c i} is an unknown scalar and z\. is the unknown vector. 

Finally, it is known in the art that infinity 
homographies between three arbitrary images Uj 9 k are related 
by the following: 

H^^HjJff^. (4) 

As noted above, point matches between any image pair 
within the image sequence 2 00 are known or may be readily 
computed (e.g., by sparse optical flow computation), and 
therefore epipoles and fundamental matrices may be easily 
determined. Furthermore, at least one infinity homography 
for an image pair (H UfX) between the first two consecutive 
images in the exemplary emobiment) within image sequence 
200 is known or may be calculated. Since computation of 
infinity homographies for two images is very difficult, 
often requiring manual intervention where decisions or 
judgments must be made (which is impractical for image 
sequences of any significant size) , the present invention 
computes all infinity homographies between any image pair 
within image sequence 200 from the point matches 2 01 and 
the known infinity homography 202. 

Equations (1) through (4) hold for any arbitrary image 
set Ujjk , so without loss of generality those variables may 
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be set as z = l, j = 2 and k = 3 (i.e., using images I 1# I 2 and 
I3) • Substituting equation (3) into equation (4) obtains: 

C \3^\3 e i3 Z \3 ~ ( C 23^23 + ^23^23 ) ^12oo • ( 5 ) 

By multiplying both sides of the equation with the matrix 
[e u ] x and noticing that [e 13 ] x e 13 =0 and [e 13 ] x i/ 13 == i^ 3 results in: 

C 13^13 = ( C 23 [ e n] x ^23 + [ e n] x e 23 Z 23 ) ^12oo ' 

which may be equivalently written as: 

C 23 [^13 ] x ^23 + [^b]* ^23 Z 23 = C 13^13^12oo * ( ^ ) 

Equation (6) is a linear equation in c 23 and z 23 which may be 
easily solved (it may be shown that the corresponding 
matrix has rank 4) . Once c 23 and z 23 are determined, 
infinity homography H 23w may be computed from equations (1) 
and (3) and infinity homography # 13o0 may be computed if 

£3 desired from equation (4) . The process may then be 

15 repeated for images I 2 , I3 and I 4 utilizing point matches 

for those images and infinity homography if 23oo , and then 

successively repeated in like manner for all remaining 
images within image sequence 200. 

Those skilled in the art will recognize that equations 
20 (1) , (3) and (4) may be employed to interpolate as well as 

extrapolate infinity homography transfer- -that is, infinity 



lOyj 



s i 
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homographies H n ^ and H 13m may be determined utilizing point 
matches and infinity homography H l3m by solving 

{c n F n )" 1 = (c 23 [e i2 ] x H 23 + [e 12 ] x e 23 z[ 3 ) H;l 

for c 23 and z 23 , then using equations (1), (3) and (4) as 

described above . 

Moreover, while the exemplary embodiment employed an 
infinity homography H X2m between the first two consecutive 
images Ii and I 2 within image sequence 2 00 and computed 
infinity homography transfers proceeding forward through 
the image sequence 200, an infinity homography for any 
consecutive image pair located anywhere within the image 
sequence 2 00 may be employed, and infinity homography 
transfers may be computed proceeding both/either forward 
and/or backward through the image sequence 200. For 
instance, if infinity homography // 23oo had been known in the 
above example rather than infinity homography H x2oo , infinity 
homographies H Uo0 and H x3ao may be determined by solving 
either 

c u F X3 = H 23a3 (c l2 [e X3 ] x H n + [e l3 ] x e X2 z T X2 ) 



for c 12 and z 12 or 



(^12^12 r = (c l3 [e n \ H l3 + [e x2 \ e X3 zf 3 )H : 



i 

23oo 
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for c n and z 13 , then using equations (1), (3) and (4) as 



This flexibility in infinity homography transfer 
according to the present invention allows an image sequence 
to be received, point matches (and epipoles and fundamental 
matrices) between image pairs computed, and any (not 
necessarily consecutive) image pair selected to compute a 
first infinity homography, which is then transferred to the 
remaining image pairs as described above. The image pair 
employed to compute the initial infinity homography may 
therefore be selected for ease in calculating that initial 
infinity homography. 

Those skilled in the art will further recognize that 
point matches, epipoles and fundamental matrices need not 
be computed for all unique image pairs within image 
sequence 200 to transfer infinity homographies according to 
the present invention. For example, point matches between 
images Ii and I 7 are not necessary unless an infinity 
homography between images Ii and I 7 is required. Since 
infinity homographies are generally only required between 
sequential images, point matches, epipoles and fundamental 
matrices need only be calculated for all unique image pairs 
for each set of three consecutive images within image 



described above. 
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sequence 2 00 (i.e., Ii, I 2 and I 3 ; I 2 , I3 and I 4 ; I 3 , I 4 and 



homographies for pairs of nonconsecutive images (e.g., H n<X} ) 
need not necessarily be calculated. 

FIGURE 3 is a high level flowchart for a process of 
computing homography transfer utilizing point matches 
according to one embodiment of the present invention. The 
process 3 00 begins with receipt of the image sequence for 
which homography transfer is to be performed (step 301) . 
Point matches for (at least selected) image pairs within 
the received image sequence are first obtained, either by 
calculation or receipt/retrieval, and epipoles and 
fundamental matrices are calculated from the point matches 
(step 302) . 

An infinity homography for one image pair within the 
image sequence is then obtained, either by calculation or 
receipt/retrieval (step 3 03) , then an image set including 
both the image pair for which the infinity homography is 
known and a third image is selected (step 3 04) . The 
selected image set will include three image pairs: the 
image pair for which an infinity homography is known (i.e., 
Ii and I 2 in the example above) ; and two "overlapping" image 
pairs (image pairs including an image from the image pair 



In-i. and I n ) . 



Similarly, infinity 
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for which the infinity homography is known, or image pairs 
I2/I3 and I1/I3 in the example above) . 

The intermediate transfer parameters of an infinity 
homography for one of the overlapping pairs is then 
computed utilizing the known infinity homography (step 
305). These intermediate parameters includes a scalar (c, y ) 
multiple for the infinity homography for that overlapping 
pair {H iJ<X) ) and a vector (z iy ) multiple for epipoles for that 

overlapping pair (e-- ). The intermediate transfer parameters 

are then utilized t.o compute the infinity homography for 
the selected overlapping image pair (step 3 06) , which may 
be used in turn to compute the infinity homography for the 
other overlapping image pair within the selected image set 
(step 307) , if desired. 

The homography transfer process (steps 3 04-3 07) is 
then repeated until all image pairs --or at least all image 
pairs for which infinity homographies are desired- -have 
been selected (step 3 08) together with an overlapping image 
pair for which an infinity homography is known. The 
process then becomes idle until another image sequence is 
received (step 3 09) for homograhy transfer according to the 
present invention. 
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The present invention allows infinity homographies for 
at least all consecutive image pairs within an image 
sequence to be computed from a single known infinity 
homography and (sufficient) point matches between the 



invention is far simpler compared to computing infinity 
homographies independently for each subject image pair, and 
is therefore extremely useful for any three dimensional 
video appl icat ion . 

It is important to note that while the present 
invention has been described in the context of a fully 
functional system, those skilled in the art will appreciate 
that at least portions of the mechanism of the present 
invention are capable of being distributed in the form of a 
machine usable medium containing instructions in a variety 
of forms, and that the present invention applies equally 
regardless of the particular type of signal bearing medium 
utilized to actually carry out the distribution.. Examples 
of machine usable mediums include: nonvolatile, hard-coded 
type mediums such as read only memories (ROMs) or erasable, 
electrically programmable read only memories (EEPROMs) , 
recordable type mediums such as floppy disks, hard disk 
drives and compact disc read only memories (CD-ROMs) or 



subject image pairs. 



The algorithm of the present 
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digital versatile discs (DVDs) , and transmission type 
"mediums such as digital and analog communication links. 

Although the present invention has been described in 
detail, those skilled in the art will understand that 
various changes, substitutions, variations, enhancements, 
nuances, gradations, lesser forms, alterations, revisions, 
improvements and knock-offs of the invention disclosed 
herein may be made without departing from the spirit and 
scope of the invention in its broadest form. 



